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ABSTRACT 

General izability theory is based upon analysis of 
variance (ANOVA) and requires estimation of variance components for 
the ANOVA design under consideration in order to compute either G 
\Generalizability) or D (Decision) coefficients. Estimation of 
variance components has a number of alternative methods available 
using SAS, BMDP, and ad hoc procedures. This study examines the 
effect of these methods on coefficeints for a large sample of 
subjects under complex designs. The Wechsler Adult Intelligence 
Scale-Revised national standardization sample provided the data base. 
Subject (n-1880) variables (age, gender, etc.) provided variety for 
the construction of complex factoral designs. Three designs were used 
to investigate variance component estimation and resultant 
g-coef f iceints. The designs included: (1) a two-factor subject x item 
design; (2) a three-factor subject x ite-n x age design; and (3) a 
four-factor subject x item x age x gender design. These designs 
allowed estimation of both within and between subject effects. 
Suggestions are made concerning equal and unequal situations. 
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Abstract 

General izabi 1 it> theory is based upon analysis of 
variance (ANOVA) and requires estimation of variance 
components for the ANOVA design under consideration in 
order to c jte either G (Genera 1 i zabi 1 ity ) or D 
(Decision) coefficients (Cardinet, Tourneur, & Allal, 
1976). Estimation of variance components has a number 
of alternative methods available using SAS, BMDP, and ad 
hoc procedures. This study examines the effect of these 
methods on coefficients for a large sample of subjects 
under complex designs. Suggestions are made concerning 
equal and unequal situations. 



Var i ance Components 

3 

ESTIMATION OF VARIANCE COMPONENTS 
USING COMPUTER PACKAGES 

I ntroduct 1 on 

The theory of general izabi 1 lty of behavioral and 
mental measurements depends upon the estimation of 
variances for various facets in the design under 
consideration, For ex amp 1 e, the si mp lest 
genera 1 izab i 1 1 ty (reliability) coefficient or 
g-coef f ic lent based upon a subj ect-by- i tem design is 
estimated using the intraniass correlation. This 
statistic estimates the ratio of subject variance to 
subject plus error variance, but it is not the oniy 
statistic that can be computed to estimate that ratio. 
Several statisticians have derived alternate methods for 
variance components estimation, inc luding Hemmer le and 
Hartley (1973) for maximum likelihood; Hartley, Rao, and 
LaMotte ( 1978 ) for the MIVQUEO method; (=iylor, Lucas, 
and Anderson '1970) for the adjusted sum of squares 
method, and various quasi-F procedures of an ad hoc 
nature. Consequently, it is not known presently how 
estimated g-statistics vary in magnitude by use of 
various estimation procedures. Recently, Be 11 (1985) 
examined several procedures within SAS and concluded 
that MIVQUEO was the most useful and efficient. 
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Although other statistical packages were not 
considered, nor were components calculated from 
unbalanced data, software probl ems concern ing the 
estimation of variance components were brought to light. 
This paper answers some of the questions raised by Bell 
and suggestions are given concerning unequal n 
s 1 tuat ions . 

The ore t ic a 1 Framework . The theory of 
general izabi 1 lty due to Cronbach, Gleser, Nanda, and 
Rajaratnam (1972) was used as the definitive work for 
all designs emp 1 oyed . Intraclass correlation 
coefficients were calculated from the variance 
components based upon devel opments by Card i net, 
Tourneur, and Allal (1976). The resulting 
g-coef f ic ients were computed according to Rentz's (i960) 
rules for calculation. 

METHOD 

The Wechsler Adult Intelligence Scale-Revised 
national standardization sample provided the data base 
(Wechsler, 1981). It consists of i860 subjects 
stratified by age, gender, race, occupation, education, 
urban-rural residence, and region of the U.S. In 
addition, several other variables were avail able, such 
as birthplace, handedness, and birth order. These 
variables provide great variety for the construction of 
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complex factoral designs. Three designs were used to 
investigate variance component estimation and resultant 
g-coef f ic 1 ents , The three designs were : (a) a two- 
factor subject x item design, (b) a three-factor subject 
x item x age design, and (c) a four-factor subject x 
item x age x gender design. These designs allowed 
estimation of both within and between subect effects. 

Variance c omponents and the resultant g- 
coefficients for the intraclass correlations were 
computed and calculated using the BMDP program 8V and 
SAS. BMDP6V uses the Cornfield and Tukey (i956) 
formulas to estimate variance components. The new SAS 
Version 5 procedure VARCOMP has four estimation methods: 
(a) TYPEi, (b) MIVQUEO, (c) ML, and (d) REML (SAS, 
1985b). The TYPEi method computes the Type i sum of 
squares for each effect (factor) and then solves the 
resulting system of equations (Gaylor, Lucas, & 
Anderson, 1970). The MIVQUEO method uses a technique 
similar to the TYPEi method except that the ri ->m 
effects are adjusted only for the fixed effects. This 
method affords a considerable timing advantage over the 
TYPEi method (SAS, 1985b). The MIVQUEO method is based 
on the Hartley, Rao, and LaMotte (197") technique 
designed to produce estimates that are locally best 
quadratic unbiased estimates. The ML (maximum-likelihood) 
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method computes max lmum- likelihood es t imates of the 
variance components by using initial MIVQUEO estimates 
and the W-tran^f ormat i on developed by Hemmerle and 
Hartley (1973). This procedure then iterates until the 
log-likelihood objective is satisfied or coverges. The 
REML (restricted maximium-1 ikel ihood ) method is similar 
to the ML method except that the likelihood is separated 
into two parts; fixed effects and random effects 
(Patterson & Thompson, 1971). Initial MIVQUEO estimates 
are iterated until convergence is met for the random 
effects part only. 

Five random samples were selected from the WAIS-R 
data hase (N=1880) to examine the stability of the 
estimates and to have both balanced and unbalanced 
designs. The WAIS-R standard nation s amp le was randomly 
divided into four subsamples (N=470 each). These four 
subsamples allowed both equal and unequal cell sizes 
according to the number of factors investigated to 
examine the effects of unequal cell sizes on estimates 
of the variance components, g~coef f ic lents, computation 
time, and memory required for computation. A fifth 
subsampl e (N=3 60 ) was created by randomly sampl ing so 
that there would be equal cell sizes for the two-, 
three-, and four-factor designs. 
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RESULTS 

Although most results were obtained for the two- 
factor solutions, no ML or REML solutions were obtained 
for the three- and four-factor designs. The time 
required for ML and REML solutions under SAS for both 
equal and unequal sample size designs became 
prohibitively expensive (over 6 minutes executed c.p.u. 
and over 15 minutes calculated c.p.u.). Bell (1985) 
also f ound the VARCOMP procedure (methods of TYPE1 and 
ML) and the GLM procedure within SAS to be prohibited by 
requiring large (over 5 minutes c.p.u. time) amounts of 
t lme and memory which are expensive. 

It was hoped by the present authors that the new 
SAS Version 5 might reduce the time and memory 
requirements using the new REML method in the VARCOMP 
procedure or the new REPEATED statement in the GLM 
procedure. The GLM REPEATED statement was inve5tigated 
using the two-factor equal cell size subsample (N=360) 
and it required over 3 minutes c.p.u. and over 2000k of 
memory. The REML method also used large amounts of time 
and memory. 

The time (in c.p.u.) and memory requirements for 
the different methods and procedures are shown in Table 
1. For the MIVQUEO and TYPE1 methods, more time and 
memory was needed as the number of factors increased. 
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Also, less time was needed for the equal cell size 
sample (EQSAM) than for the other four samples for each 
design (two-, three-, and four-factors). This should be 
true regardless of balanced versus unbalanced designs 
because there were less subjects in EQSAM compared to 
the other four subs amp les . 

The BMDP program 8V provided the most efficient 
estimates with balanced data. Interestingly, tlie BMDP8V 
program was even more efficient when accessed through 
SAS (SAS, 1985a). This finding was important because 
the data must be sorted carefully for data entry to be 
used in PMDP8V according to slowest and fastest moving 
indexes or levels of factors (Dixon, 1985). The data 
can be sorted easily with the SORT procedure prior to 
the BMDP procedure in SAS. Also, this finding allows 
for the analysis of SAS structured data sets. 

Table 2 shows the estimated g-coef f ic lents for the 
various procedures and methods, samples, and designs. 
For the two-factor solutions there was no variation to 
the third decimal place across procedures for EQSAM. 
The estimates varied from .908 to .931 across all five 
random samples. Since reliability estimates are often 
reported to two decimal places, the .91 to .93 
differ ence is negligible. 

The three-factor solutions provided est imates that 
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were very close to the two-factor solutions. These 
estimates varied from g-coef f 1 c lents of .907 to .935. 
Although these estimates varied a little more than the 
two-factor solutions, the estimates were very similar 
across procedures and sampl es . 

The four-factor solutions revealed more variation 
across procedures and samples than either the two- or 
three-factor solutions. This was due in large part to 
the MIVQUEO method consistently estimating larger error 
variances. The procedures varied from .920 to .926 for 
EQSAM, but varied from .690 to .937 across all five 
random samples. Even these variations may not represent 
substantial differences in terms of efficiency or 
prec i s i on . 

CONCLUSIONS 

The major results of this study supplement Bell's 
(1985) results. For typical, large test data sets it 
will be extremely expensive to estimate g-coef f ic lents 
if TYPE 1, ML, or REML methods of SAS's VARCOMP 
proc edure are used to compute var i an ce components . 
BMDP, while restricted to equal cell size designs, was 
efficient in its computations and required fractions of 
the time and memory used by those three methods. For 
many situations unbalanced designs will occur, however, 
with excessive loss of information accompanying 
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balancing. This situation can be assessed by using the 
MiVQUEO method of SAS's VARCCMP procedure for the 
unequal cell size sampl e and then comparing these 
estimates with BMDP6V estimates of the reduced, balanced, 
equal cell size samp le. It is un likely that these two 
methods will give drastically different estimates for 
simple designs (two- or three-factors), but different 
estimates couli result for designs with many factors and 
many l eve 1 s . 

It was anticipated that the new SAS Version 5 would 
help meet the need for efficient compulation of within 
and between subject effects with the REPEATED statement 
of the GLM procedure. This has lowered the cost for 
repeated measures analysis, but is not efficient as 
applied to the calculation of variance components from 
Type i expected mean squares for genera 1 1 zab i 1 i ty 
theory. 
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Table 1 



C.P.U. and Memory Requirements 



Method 


Fac tors 
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3000 K ! 




3 


over 360 
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3000 K : 


ML 


2 


over 360 


3000 K ! 
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3000 k : 
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3000 K : 


3MDP 


2 
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600 K ! 
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1 . 5 
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1 . 9 
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2 


0 . 4 
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0 . 8 


600 K : 
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600 K : 



* NOTE: Equal cell size sample (EQSAM) requirements 
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Table 2 

G-coefficient Estimates 

Two-factor Solutions 



Procedure 
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One 
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Three 
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. 934 : 



d 1 cr 

ERIC 1 5 



